bert_tokenizer 0.1.3

This crate is a Rust port of Google's BERT WordPiece tokenizer.
Documentation

bert_tokenizer

[ API doc | crates.io ]

The crate provides the port of the original BERT tokenizer from the Google BERT repository.

License

MIT license. See LICENSE file for full license.